DOCUMENT RESUME 



ED 270 499 

AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENCY 

REPORT NO 
PUB DATE 
CONTRACT 
NOTE 

PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



TM 860 389 

Reckase, Mark D. 

The Use of Tailored Testing with Instructional 
Programs, Final Report, 

American Coll, Testing Program, Iowa City, IA. Test 
Development Div. 

Office of Naval Research, Arlington, Va. Personnel 
and Training Research Programs Office. 
ONR-86-1 
Mar 86 

NOOO14-82-K-0716 
43p. 

Reports - Research/Technical (143) — 
Tests/Evaluation Instruments (160) 

MF01/PC02 Plus Postage. 

♦Achievement Tests; *Adaptive Testing; Attitude 
Measures; "Computer Assisted Testing; Latent Trait 
Theory; Microcomputers; "Military Training; 
Postsecondary Education; Radar; "Student Attitudes; 
"Test Format; Testing Problems 

Computerized Testing Evaluative Survey; Paper and 
Pencil Tests 



A computerized testing system was implemented in 
conjunction with the Radar Technician Training Course at the Naval 
Training Center, Great Lakes, Illinois, The feasibility of the system 
and students 1 attitudes toward it were examined. The system, a 
multilevel, microprocessor-based computer network, administered tests 
in a sequential, fixed length format for the purpose of comparing the 
effect of mode of administration of test items (computerized versus 
paper-ard-pencil) . Since the one-parameter and three-parameter 
logistic models were both determined to be acceptable, the less 
expensive one-parameter model was used. The results indicated that, 
in spite of hardware problems, the system design was viable and that 
microcomputers could be used to administer achievement tests in an 
instructional program environment. Some items were found to operate 
differently when administered on a computer screen as compared to a 
paper- and- pencil administration. Further research was suggested to 
determine the cause of the differences in item performance. A 23-item 
attitude survey, which is appended, was administered to a sample of 
136 trainees. The trainees generally had no difficulty taking the 
test on the computer r and 62 percent indicated that they enjoyed it. 
(Author/GDC) 



********************************************************************** 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 

******* ***** ************************ t ********************************* 



Final Report 



The Use of Tailored Testing 
With Instructional Programs 

Mark D. Reckase 



Research Report ONR 86-1 
March 1986 



U S DEPARTMENT OF EDUCATION 

Ottice of Educational Research and improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

L*Jh > document has been reproduced as 
( rWeived from the person or organt/atton 
originating it 

^ Minor changes have been made to tmprove 
reproduction quality 



• Homts ot vtf>w or opinions stated '0 thts docu 
ment do not necessarily represent official 
OERl position or pohcv 




The American College Testing Program 
Assessment Programs Area 
Test Development Division 
Iowa City. Iowa 52243 



Prepared under Contract No. N00014-82-K-0716 
Contract Authority Identification Number NR1 50-499 
with the Personnel and Training Research Programs 
Psychological Sciences Division 
Office of Naval Research 

Approved for public release, distribution unlimited 
Reproduction in whole or in part is permitted for 
any purpose of the United States Government 



2 



\CLASSI: ltD 



SEC~ 



iLAbs-F ca:.u,\ o- "-is =>a':e 



REPORT DOCUMENTATION PAGE 



la REPORT ScCJRtT/ ClASSiFiCAT.CN 
UNCLASSIFIED 



2a SECURITY CLASS F-CATiON AUTHORITY 



2b DEC-ASS.f "CATION / DOWNGRADING SCHEDULE 



4 PERFORMING ORGANIZATION -EPORT NUMBER'S) 

ON'R 86-1 



6a NAME OF PERFORMING C»C AMI A HON 

ACT 



OFF CE SYMBOL 
(tf applicable) 



6c AD ORE S3 Cry, Srace, jno ZlPCoae) 

P.O. BOX 163 

IOWA CITY, I A 52243 



8a NAME OF FUNDING / SPONSORING 
ORGANIZATION 



3b OFF'CE SYMBOL 
(If applicable) 



8c ADDRESS (City, State, and ZIP Code) 



io RESTRICTIVE MARKINGS 



3 DISTRIBUTION/ AVAILA8ILITY OF REPORT 

Approved for public release: distribution un- 
limited. Reproduction in whole or in part is 

permitted for snv Purpose of r'^e U.S. • 



5 MONITORING ORGANIZATION REPORT NUM6ER(S) 



7a NAME OF MONITORING ORGANIZATION 

PERSONNEL & TRAINING RESEARCH PROGRAMS 
OFFICE OF NAVAL RESEARCH 



7b ADDRESS (C/fy, State, and ZfPCode) 

Personnel & Training Research Programs 
Office of Naval Research (Code 442PT) 
Arlington, VA 22217 



9 PROCUREMENT .NSTRUMENT 'DENTIFlCATlON NUMBER 

N00O14-82-K-07L6 



10 SOURCE OF FUNDING NUMBERS 



PROGRAM 
ELEMENT NO 

61153N 



U TITLE (inciuae Security Classification) 

The Use of Tailored Testing with Instructional Programs 



PROJECT 
NO 

RR042-04 



TASK 
NO 

042-04-01 



WORK UN II 
ACCESS.ON ISO 

NR150-499 



12 PERSONAL AUTHOR(S) 

Mark D. Reckase 



!3a TYPE OF 


REPORT 


1 :d time covered 


14 DATE OF 


REPORT (Year, Month, Day) 


15 PAGE COUNT 


Final 


Report 


^OM8?SF?m tors AUG 11 


1^86, 


March 


31 



16 SUPPLEMENTARY NOTATION 



17 



COSATI CODES 



FlELO 


GROUP 


SUB-GROUP 















18 SUBJECT TERMS (Continue on reverse if necessary and identify by block number) 
Computerized testing Item response theory 

Computerized adaptive testing Mode effect 



19 ABSTRACT {Continue on reverse if necessary and identify by block number) 

This report describes the computerized testing system fiat was impl emented in con- 
junction with the Radar Technician Training Course at the Na/al Training Center at Great 
Lakes, Illinois and the research that was performed using the system. The system was a 
multilevel, microprocessor-based computer network with each testing station capable of 
operating as a stand-alone test administering computer. The system was used to administer 
tests in a sequential, fixed length format for the purpose of gaining information on the 
effect of mode of administration on test items. 

The project demonstrated that the system design was viable and that microcomputers 
could be used to administer achievement tests in an instructional program environment. 
Trainees were found to have no difficulty taking the test on the computer terminals. For 
the small item pools available for this course, a one-parameter logistic based testing 



20 DISTRIBUTION /AVAILABILITY OF ABSTRACT 

□ UNCLASSIFIED/UNLIMITED H SAME AS RPT □ OTIC USERS 


21 ABSTRACT SECURITY CLASSIFICATION 

UNCLASSIFIED 


22a NAME OF RESPONSIBLE INDIVIDUAL 

ffo Charles Davis 


22b TELEPHONE (include Area Code) 

(202) 696-4046 


22c OFFICE SYMBOL 



OERICI 1473 ,84 MAR 83 APR edition may be used until exhausted SECURITY CLASSIFICATION OF THIS PAGE 

h n ii mimnriT i niin All other editions are obsolete — — 
Q UNCLASSIFIED 



19. 

procedure was found to operate adequately. Some items were found to operate differently 
when administered on a computer screen as compared to a paper-and-pencil administration. 
Further research is needed to determine the cause of the differences in item performance 



Contents 



Page 



Introduction # 2 

Instructional Environment 2 

Course Description , 3 

Course Exams 11 

Systems Design , 13 

Hardware 13 

Software 16 

Research Projects 17 

Summary and Conclusions 22 

References , , .24 

Appendix 25 



ERLC 



5 



2 



Final Report 

The Use of Tailored Testing with Instructional Programs 

The primary objective of the project was to investigate issues that are 
related to the Implementation of tailored, or adaptive, testing in the 
instructional programs environment. These issues are of tvo major types: 

• Those related to the design of a computer system f. the 
administration, scoring, and reporting of results in tailored 
tests . 

• Those related to the psychometric theory that is the foundation 
of tailored testing. 

To address these issues in a practical and realistic way, this project 
involved the staff at the Great Lakes Naval Training Center in developing and 
testing a computer system for use in administering tests in coniunction with 
instruction orograms at the base. This report is a summary of the knowledge 
gained from the development and implementation of the system. 

The report is composed of three sections. The first describes the 
instructional environment at the training center that served as a test site 
for the tailored testing system. The second section describes the computer 
hardware used for the project and the software that was developed for the 
implementation of instructional testing at Great Lakes Naval Training 
Center. The third section describes the psychometric research that was 
performed as part of the project and summarizes the results of the research. 

Instruct ional Environment 
Since the goal of thi^ project was to evaluate tailored testing for 
instructional purposes in a realistic educational environment, arrangements 
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were made to implement tailored testing in the Radar Technician Training 
Course in the Electronics Technician School at the Naval Training Center at 
Great Lakes, Illinois. This course was selected for use because it was taken 
by a relacively large number of trainees (approximately 60 new trainees 
started the course each week), it had fairly extensive item pools available 
for the material covered by the course, the trainees were fairly sophisticated 
about computer equipment, and systems programming support was available from 
the Naval Education and Training Program Development Center Detachment at 
Great Lakes. 

Course Description 

The Radar Technician Training Course is a six week course that is divided 
into three major areas. These areas, in turn, are subdivided into 
instructional modules. Table 1 presents a brief outline of the course, 
showing the major areas and tiie modules. The performance of trainees in the 
course was evaluated using examinations covering the three major areas: 
power, transmitter, and receiver. The trainees were required to receive a 
score equal to or greater than 64% of the total possible on each exam in order 
to proceed through the course. If they failed a test they were given 
remediation and retested over the same material, but with an alternate form of 
the test. Trainees who failed an examination three times were dropped from 
the course. 

Although the testing and remediation procedure implied a self-paced 
instructional strategy, in actuality there was little flexibility in the r^te 
at which trainees could proceed through the course. At most, trainees were 
allowed co spend three extra days on the material from a section of the 
course. 

o 1 
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Table i 
Course Outline 
for the Radar Technicians Training Course 



Ma ior Module 

£rea Number Description 



4.1.1 


Introduction to 


AN/SPS-10 Radar Set 


Power 4.1.2 


Primary Power Distribution 


':.1.3 


AC Voltage Regulator 


4.1.4 


DC Power Supply 




4.2.1 


Modulator 




Transmitter 


4.2.2 


Radar Transmitter 


4.2.3 


RF System 




4.3.1 


Radar Receivers 




Receiver 


4.3.2 


Adapter , Indicator 


4.3.3 


Video Clutter Suppressor 


4.3.4 


Antenna System 





The instructional schedule for the course is given in Table 2. This 
table gives the topic number for the class, the type of class, the class 
period, the topic description, and the ratio of students to instructor. The 
first three digits of the topic number refer to the instructional modules 
listed in Table 1. The fourth digit refers to specific topics within the 
module. The class period indicates the number of hours since the beginning of 
training. The Radar Technician Training Course begins at hour 741 with a 
three hour class on "introduction to Radar Systems" and ends at hour 980 after 
the four hour performance test on receivers. Thus, the course is 240 hours 
long, arranged into 30, eight hour days. 
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Table 2 

Advanced Electronics Field Master Schedule 
Radar Technicians Training Course 



TOPIC NO. 



TYPE 



PERIOD 



TOPIC 



RATIO 



4.1.1.1 



4.1.1.1 



4.1.1.2 



4.1.1.2 



4.1.1.3 



4.1.1.3 



4.1.1.4 



Class 

Class 
Class 

Class 

Class 
Class 

Class 



Clasi 



Class 
Lab 



Clasj; 



741 
742 
743 
744 

745 
746 
747 
748 
749 
750 
751 
752 

753 
754 
75b 
756 
757 
758 
759 
760 

761 
762 
763 
764 
765 
766 
767 
768 

769 
770 
771 
772 
773 
774 
775 
776 



Introduction to Radar Systems 



GMT 



Intro to Radar Sys (cont'd) 



Radar Principles 



GMT 



Radar Principles (cont'd) 



Introduction to AN/SPS-10 



Intro to AN/SPS-10 



GMT 

AN/SPS-10 Lab 

4.1.1.4 1 J Fami liarization 
with the AN/SPS-10 Radar Set 
4.1.1.4 2J AN/SPS-10 Radar Set 
and AN/SPA-25 Indicator 
operating procedures 

GMT 



20:1 
20:1 
20:1 



20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 



20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 

20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 

6:1 
6:1 
6:1 
6:1 
6:1 
6:1 
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Table 2 (Continued) 



TOPIC NO. 



TYPE 



PERIOD 



TOPIC 



RATIO 



4.1.2.1 



Class 



4.1.2.1 



4.1.3.1 



4.1.4.1 



4.1.2 - 4.1.3 



4.1.2.2 



4 1.3.2 

4.1.3.2 
4.1.4.2 



4.1.2.2 



Class 
Class 



Class 
Class 

Class 



Lab 



Lab 



Class 
Class 



Lab 



777 
778 
779 
780 
781 
782 
783 
784 

785 
786 
787 
788 

789 
790 
791 
792 

793 
794 
795 
796 
797 
798 
799 
800 

801 
802 
803 
804 
805 
806 

807 
808 
809 
810 
811 
812 
813 
814 
815 



Primary Power Distribution 



GMT 

Pri Power Dist (cont'd) 



GMT 

AC Voltage Regulator 
DC Power Supply 



AC Voltage Regulator Lab 
DC Power Suppl y Lab 

Primary Power Distribution Lab 

4.1.2.2 1J 
GMT 

Written Exam (Power) 



Primary Power Distribution Lab 
4.1.2.2 2J 



20: 1 
20:1 
20: 1 
20: 1 
20: 1 
20: 1 
20: 1 



20:1 
20: 1 
20:1 
20: 1 

20:1 
20:1 
20:1 



20: 1 
20: 1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 

6:1 
6:1 
6:1 
6:1 
6:1 
6:1 

6:1 

20:1 
20: 1 
20:1 
20: 1 
6:1 
6:1 
6:1 
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Table 2 (Continued) 



TOPIC NO. TYPE PERIOD TOPIC RATIO 



Class 816 GMT 

A. 1.2.2 Lab 817 Primary Power Distribution Lab 6:1 

818 4.1.2.2 2J (cont'd) 6:1 

819 6:1 

820 6:1 
£21 6:1 

822 6:1 

823 6:1 
Class 824 GMT 

4.1.2 - 4.1.3 Lab 825 Performance T es t (PPD/ACVR) 6:1 

826 6:1 

827 6:1 

828 6:1 
4.2.1.1 Class 829 Vacuum Tube TPG 20:1 

830 20:1 

831 20:1 
Class 832 GMT 20:1 

4.2.1.1 Class 833 Vacuum Tube TPG (cont 1 d) 20:1 

834 20:1 

835 20:1 

836 20:1 

837 20:1 

838 20:1 

839 20:1 

840 20:1 

4.2.1.2 Class 841 Modulator (MPG) 20:1 

842 20:1 

843 20:1 

844 20: 1 

845 20:1 

846 20:1 

4.2.1.4 Class 847 Solid State TPG 20:1 

Class 3 'i8 GM1 

4.2.1.4 Class 849 Solid State TPG (cont'd) 20:1 

850 20:1 

851 20:1 

852 20:1 

4.2.2.1 Class 853 Microwave Devices 20:1 

854 20; 1 

855 20:1 
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Table 2 (Continued) 



TOPIC NO. 



TYPE PERIOD TOPIC 



RATIO 



4.2.2.1 



4.2.2.2 



4.2.2.3 

4.2.3.1 
4.2.1.3 

4.2.1.3 



4.2.2.4 



4.2.3,2 



Class 
CI ass 

Class 



Class 
Class 



Class 

Lab 
Class 

Lab 



Lab 



Lab 



Class 

4.2.1 - 4.2.2 Class 
4.2.1 - 4.2.2 Lab 

Class 



856 

857 
858 
859 
860 
861 
862 
863 
864 
865 
866 
867 
868 
869 
870 
871 
872 

873 
874 
875 
876 
877 
878 

879 
880 
881 
882 
883 
884 
885 
886 
887 
888 

889 
890 
891 
892 
893 
894 
895 
896 



GMT 



Microwave Devices (cont'd) 



Microwave Xmtr Tubes 



GMT 

Transmitter 



RF Systems 

TPG and MPG Lab 4.2.1.3 1J 
GMT 

TPG and MPG Lab 4.2.1.3 2 J 



Transmitter Lab 4.2.2.4 1J 

Transmitter Lab (cont'd) 
4.2.2.4 2J & 3J 

RF System 4.2.3.2. 1J 



GMT 

Written Exam 



(Xmtr) 



Performance Test (Xmtr) 



GMT 



20:1 

20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 

20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
6:1 



6:1 
6:1 
6:1 
6:1 
6:1 
6:1 

6:1 
6:1 
6:1 
6:1 
6:1 
6:1 
6:1 
6:1 
6:1 



20:1 
20:1 
20:1 
6:1 
6:1 
6:1 
6:1 
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Table 2 (Continued) 



TOPIC NO. 



TYPE 



PERIOD 



TOPIC 



RATIO 



4.3.1.. 
4.3.1.1 



Class 
Class 



4.3.1.2 

4.3.1.3 
4.3.1.5 



4.3.1.4 



4.3.2.1 

4.3.2.2 
4.3.2.2 

4.3.3.1 
4.3.4.1 



4.3.4.1 
4.3.4.2 



4.3.4.3 



Class 

Class 

Class 
Class 

Class 
Class 



Class 

Class 
Class 

Class 

Class 

Class 

Class 
Class 

Lab 



897 

898 
899 
900 
901 
902 
903 
904 

905 
906 
907 
908 
909 
910 
911 
912 

913 
914 
915 
916 
917 
918 
919 
920 
921 

922 
923 

924 
925 
926 
927 
928 

929 
930 
931 
932 
933 
934 



Introduction to Receivers 
Introduction to Receivers (cont'd) 



GMT 

IF Strip 

Detectors am' Video Circuits 
Automatic Frequency Control 



GMT 



Interference Elimination Circuits 



Adapter Indicator Trigger Ckts 

Adapter Indicator Video Ckts 
Adapter Indicator Video Ckts 

Video Clutter Suppressor 
Synchros/Servos/Antennas 
GMT 

Synchros /Servos /Antennas 
Antennas System? 



Antenna System Lab 
4.3.4.3 U & 2J 



20:1 

20:1 
20:1 
20:1 
20:1 
20:1 
20:1 



20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 



20: 1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 
20:1 

20:1 
20:1 

20:1 
20:1 
20:1 
20:1 



20:1 
20:1 
20:1 
20:1 
6:1 
6:1 
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Table 2 (Continued) 



TOPIC NO. TYPE PERIOD TOPIC RATIO 



935 6:1 

Class 936 GMT 6:1 

4.3.1.6 Lab 937 Receiver Lab 4.3.1.6 1J 6:1 

138 6:1 

939 6:1 

940 6:1 

941 6U 

942 6:1 

943 6:1 

944 GMT 6:1 

4.3.1.6 Lab 945 Receiver Lab (cont'd) 6:1 

946 6:1 

947 6:1 

948 6:1 

949 6:1 

4.3.2.3 Lab 950 Adapter Indicaor Lab 6:1 

951 6:1 

Class 952 GMT 

4.3,2.3 Lab 953 Adapter Indicator Lab (cont'd) 6:1 

954 6:1 

955 6:1 
Lab 956 Overall Systems Labs 6:1 

957 6:1 

958 6:1 

959 6:1 

960 6:1 
Lab 961 Overall Systems Lab (cont'd) 6:1 

962 6:1 

963 6:1 

964 6:1 

965 6:1 

966 6:1 

967 6:1 
Class 968 GMT 

Class 969 Area Review 20:1 

970 20:1 

971 20:1 
4.3.1 - 4.3.2 Class 972 Written Exam (Rcvr) 20:1 

973 20:1 

974 20:1 
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Table 2 (Continued) 



TOPIC NO. 



TYPE 



PERIOD 



TOPIC 



RATIO 



975 
976 



20:1 



Class 



GMT 



4.3.1 - 



4.3.2 



Lab 



977 
978 
979 
980 



Performance Test (Rcvr) 



6:1 
6:1 
6:1 
6:1 



Course Exams 

The three tests used for this project were administered at hour 809 
(Power), 889 (Transmitter), and 972 (Receiver). Two forms of each of the 
tests were available for use. Table 3 summarizes the characteristics of each 
of the tests. The tests are quite variable in quality and some have fairly 
low reliabilities. Each of the tests was constructed by selecting the items 
from an item pool that had been developed for each content area. A 
description of the item pool for each of the course consent areas is given in 
Table 4. Since the items were sampled from the item pool with teplacement , 
the test forms frequently had items in common and not all items in the pool 
were used. Therefore, the number of items available for calibration using 
item response theory models was less than the total number of items in the 
pool. The fifth column of the table indicates the nimber of items available 
for calibration and the sixth column gives the sample size for calibration. 
For tests A22 and A23, the sample sizes varied for the items within the pool 
oecause of overlap in the tests. 
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Table 3 

Characteristics of Coarse Examinations 



Examinat ion 


Number of 
I terns 


X 


SD 


XR-20 


P 


pc • bi s 


N 


A211 


43 


34.12 


5.34 


.83 


.79 


.38 


497 


A212 


44 


32.15 


3.41 


.51 


.73 


.13 


428 


A221 


30 


23.88 


4.05 


.76 


.80 


.32 


515 


A222 


30 


25.83 


2.30 


.50 


.86 


.12 


415 


A231 


30 


24.24 


2.80 


.52 


.81 


.12 


448 


A232 


30 


23.67 


2.79 


.49 


.79 


.13 


410 



Note: The first two digits of the examination code indicate the course 
component and the last digit indicates the form. 
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Table 4 

Radar Technicians Training Course 
Item Pool 



Test 


Module 


Number of Items 


Total 


Number of 
Item Calibrated 


N 


A21 


1 


60 










2 


34 










3 


17 










4 


8 














119 


86 


1160 


A22 


1 


52 










2 


34 










3 


11 














97 


60 


400-700 


A23 














1 


40 










2 


10 










3 


1 










4 


27 














78 


60 


400-700 



System Design 



Hardwar e 

The system design for the computerized adaptive testing project di Great Lakes 
was predicated on several assumptions. They were: 

1. Each testing station should be capable of functioning as a 
stand-alone test administrator for reasons oT system 
reliability. 

2. The results of a test would have to be accumulated for a 
class so that class reports and item analyses could be 
generated. 

3. The system should be able to administer different tests to 
different students . 

17 
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In order to meet the assumptions with equipment that was available at the 
start of ^he project, a hierarchical computer structure was designed. The 
system had three levels: (l) the testing stations, (2) midlevel computers for 
collecting results from eight testing stations and sending them to a central 
processor, and (3) a large, top level computer that would compile data and 
generate the reports. This configuration is shown schematically in Figure 1. 

Each testing station consisted of an Ohio Scientific Challenger computer 
with two floppy disk drives and the OS 65U VI. 3 operating system and a 
Hazeltime 1420 terminal. The terminals were located in carrels in a testing 
room. The computers were located in a separate room that was under the 
control of the instructional staff. 

Eight of the testing stations were connected to each midlevel compu er. 
This computer was also an Ohio Scientific Challenger computer with two floppy 
disk drives and the OS 65U VI. 3 operating system. Although four midlevel 
computers were in place at Great Lakes, only 22 testing stations were 
available. Thus, the full 32 testing station capacity of the system was not 
realized . 

The four midlevel computers transmitted test results to the top level 
computer. This computer was an Ohio Scientific C-3B microprocessor with 48K 
random disk and a 74 megabyte (formated) Winchester hard disk. The computer 
also had two floppy disk drives and a tape backup. Two of these computers 
were purchased for the project. One served as a backup for the other to 
insure system reliability. The two top level computers also allowed for 
potential expansion to 64 testing stations. 
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Software 



Two test administration programs were developed to run on the system. 
The first was a program that administered a fixed set of test items to each 
examinee with a randomly selected order. That is, each examinee received the 
same set of test items, but the order of the items was randomly determined for 
each individual. This program was to be used at the initiation of 
computerized testing to collect data addressing the issue of whether items 
function the same when administered cn a computer screen as when administered 
in paper-and-pencil form. The fixed set of items administered by the computer 
was the same item set used on the regular paper-and-pencil tests. 

The second administration program developed during the project was for 
the adapt've administration of the course tests. This program was based on 
the one-parameter logistic (Rasch) item characteristic curve model. This 
model was selected as a result of a research study that will be described 
later in this report. 

Adaptive testing procedures require algorithms for selecting items, for 
estimating ability, and for terminating the testing session. The program 
produced for this project used maximum information item selection, maximum 
likelihood ability estimation, and a sequential probability ratio test 
(Reckase, 1983) for the pass/fail decision to terminate testing. Since the 
maximum likelihood estimation procedure cannot compute ability estimates until 
both correct and incorrect responses have been obtained, a fixed stepsize, up- 
and-down procedure was used to obtain ability estimates before both types of 
responses were available. 

After the final ability estimate was obtained for each examinee, the item 
response theory theta estimate was converted to an estimated true-score based 
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on the entire item pool for that test. The estimated true score was the score 
reported to both the examinee and the instructor. 

Both of the test administration programs were written in Ohio Scientific 
BASIC for use with the OS 65U VI. 3 operating system. They both accessed item 
pools stored on floppy disks at the testing station computers. 

System software for the communications among computers was also developed 
as part of the project. Personnel from the Naval Education and Training 
Program Development Center Detachment at Great Lakes assisted with the 
development of this software. This set of programs accumulated test results 
from the testing stations and stored them on the hard disk at the top level 
computers. Other software developed on the project was used to generate 
reports for the instructor ujing this information. 

The computer system used for administering the test at Great Lakes was 
described in several professional papers during the life of the project. The 
references for these papers are given below. 

McKinley, R.L., & Reckase, M.D. (1983). Adaptive testing in a military 
training environment. In Proceedings of the 25th Ann ual Conference o f the 
Military Te s ting Association (pp. 118-123). Pensacola, Florida: Naval 
Education and Training Program Development Center, Sanfley. 

McKinley, R.L., & Reckase, M.D. (1984, April). Implementing an adaptive 
testing program in an instructional program environment . Paper presented at 
the meeting of the American Educational Research Association, New Orleans. 

Reckase, M.D., & McKinley, R.L. (1984, August). The use of adaptive testing 
for instructional programs . Paper presented at the meeting of the American 
Psychological Association, Toronto. 



Three studies were performed as part of this research project. They 
vere: (a) a study to gain information needed to select the appropriate item 
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response theory model for use with the ite r n pools available for the project, 
(b) a study to determine che effect of mode of t^.st administration (paper-and- 
pencil or computer) on the operation of the test items, and (c) a survey of 
the attitudes of trainees toward taking tests on a computer. 

The results of the first study, on the appropriate item response theory 
model for adaptive testing using the radar technician item pool, were reported 
in a technical report and a convention paper. The references for the report 
and paper are given below. 

McKinley, R.L., & Reckase, M.D. (1983). An evaluation of one-and three- 
parameter logi st ic tailored te sting procedures for use with small item 
pools . (Research Report ONR 83-1). Iowa City, Iowa: The American College 
Testing Program. 

McKinley, R.L., & Reckase, M.D. (1984, April). An evaluation of one-and 
three-parameter logistic tailored testing procedures for use with small item 
pools . Paper presented at the meeting of the National Council on Measurement 
in Education, New Orleans. 

This study used a two-stage evaluation plan to compare the ability 
estimates yielded by computerized adaptive test ; ng procedures based on the one 
parameter logistic (1PL) and the three parameter logistic models (3PL). The 
first stage of the study used real data, while the second stage used simulated 
data. In the first stage, response data for 3000 examinees were obtained J or 
the 40 items on a form of the ACT Assessment Mathematics Usage Test. The 
first 2000 cases were used to obtain item parameter estimates for both 
models. Using these ertimates, 1PL and 3PL tailored tests were simulated 
using the response data for the remaining 1000 cases. Both computerized 
adaptive testing procedures employed maximum likelihood ability estimation and 
maximum information item selection. The sets of ability estimates obtained 
from the two procedures were then compared. 
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In the second stage of the study, response data for 3000 cases were 
simulated using the 3PL item parameter estimates from the fir^t stage as true 
parameters. True abilities were selected for the simulation from the standard 
normal distribution. The first 2000 cases of the generated data were used for 
the 1PL and 3PL calibration of the items, and the remaining 1000 cases were 
used to simulate 1PL and 3PL adaptive tests. The ability estimates obtained 
from the two procedures were compared to each other and the true ability 
parameters . 

The results of both stages of the study indicated that the 1PL and 3PL 
adaptive tests yielded highly correlated ability estimates, and there was not 
apparent advantage in terms of ability estimation to using one of the models 
over the other. Because the 1PL procedure was less expensive to use, it was 
the recommended model for this application. 

The result of the second study, on the effect of mode of tesi 
administration, was reported in a convention paper. The reference for the 
paper is given below. 

Ackerman, T.A. (1985, October). An investigation of the eifect of 
administering test items via the computer . Paper presented at the meeting of 
the Midwest Educational Research Association, Chicago. 

This study compared the performance of 86 items from the item pool for 
the Radar Technician Training Course when administered on a pape.-and-penci 1 
test to that when administered on a computer screen. The responses to the 
items were first analyzed using the one-parameter logistic mcdel to obtain 
difficulty parameter estimates from each of the administrations. The pairs of 
estimates for the items were then analyzed using a principal components 
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technique to determine which items had difficulty parameter estimates that did 
not fall along a simple linear function. 

Of the 86 items analyzed, 26 were found to exhibit significant 
differences in performance related to administrative medium. Eleven items 
were found to be harder when administered on the computer screen, and 15 were 
found to be easier. No obvious reasons could be determined for the 
differences. Several hypotheses, such as the differences being related to 
amount of verbal material in an item or item format were eliminated. Further 
work needs to be done to determine the cause of the "mode effect". 

The third research study carried out as part of this project was a survey 
of the attitudes of trainees at Great Lakes toward Mie administration of tests 
by computer. A sample of 136 trainees was administered an attitude survey 
immediately following the administration of a test to them by computer. A 
copy of the survey is included in an appendix to this report. The 
distribution of responses to the 23 items in the survey is presented in 



Table 5. 
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Table 5 
Distribution of Responses 
to the computerized Testing Survey 



Item Response 

abed 

1 16 AO 56 25 

2 7 12 80 38 

3 124 13 

4 1 4 132 

5 30 20 84 

6 75 13 48 

7 7 78 52 

8 2 20 1 113 

9 3 12 23 98 

10 118 13 4 

11 115 18 3 

12 21 115 

13 13* 1 

14 56 64 16 

15 26 92 18 

16 3 129 4 

17 1 132 3 

18 54 67 14 

19 135 1 

20 32 JO 33 

21 85 26 24 

22 93 31 11 

23 12 114 9 
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The responses generally show that the trainees had no difficulty taking 
the test on the computer and, in fact, 622 indicated that they "enjoyed" the 
computerized test. They felt that the instructions to the test were clear and 
that the pacing of the test was not too fast (the items were untimed), 
although they indicated that taking the test on the computer was faster than 
for a paper-and-pencil test. There was little indication of problems with 
reading the terminal screen or with eye fa "gue. The trainees had no 
difficulty finding the proper keys on the terminal keyboard. The only 
negative comment concerned examinees being unable to go back to questions once 
they were off the screen. Sixty-eight percent indicated that they were 
bothered by being unable to go back to previous items. Overall, the trainees 
had no difficulty with the computerized test administration. 



This report describes the computerized testing system that was 
implemented in conjunction with the Radar Technician Training Course at the 
Naval "raining Center at Great Lakes, Illinois and the research that was 
performed using the system. The system was a multilevel, microprocessor-based 
computer network with each testing station capable of operating as a stand- 
alone test administration system. The system was used to administer tests in 
a sequential, fixed length format for the purpose of gaining information on 
the effect cf mode of administration on test items. 

Although software was developed for an adaptive administration of test 
items, the system was never used for adaptive testing because of repeated 
hardware failures. These failures were particularly vexing because the 
hardware manufacturer, Ohio Scientific Inc., went out of business during the 
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project, and maintenance was difficult to obtain. The computer system used is 
now obsolete and cannot be obtained through any source. The hardware problems 
emphasize the need for developing £ computerized testing system using hardware 
from a well established manufacturer. 

Despite the hardware difficulties, the project did demonstrate that the 
system design was viable and that microcomputers could be used to administer 
achievement tests in an instructional program environment. The survey 
research showed that the trainees had no difficulty taking the test on the 
computer terminals. The research on model selection showed chat, for the 
small item pools available for this course, a one-parameter logistic based 
testing procedure would perform adequately. 

One research finding from this project indicated that some caution should 
be exercised when implementing computerized testing. It was found that some 
test ite:ns performed differently when administered on a terminal screen than 
when administered on a paper-and-penci 1 test. Further research is needed to 
determine the kinds of items that are affected by mode of administration and 
whether the effect will make the item easier or harder than the paper-and- 
pencil form. 

Overall, this project has shown that computerized testing can be 
effectively used, but that system reliability 3hould be a major factor in the 
development of such a system. With the recent advances in microcomputer 
hardware, widespread implementation of computerized testing in support of 
instruction can soon be expected. 
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PLEASE COMPLETE THE FORM BELOW AND RETURN IT TO THE TEST ADMINISTRATOR bEFCRE 
LEAVING THE TESTING ROOM. 

NAM E 

LAST FIRST MIDDLE INITIAL 

(Please Print ) 

SOCIAL SECURITY NUMBER 



DATE 

CLASS NUMBER SHIFT 
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QUESTIONNAIRE 



We are interested in your reactions to this new form of testing. Your answers 
to these questions will provide useful information about this testing 
process. Please read each question carefully before responding. 

CIRCLE THE LETTER OF THE APPROPRIATE ANSWER TO EACH ITEM. 

1. I have used a computer. . r 

a. never before. 

b. once or twice. 

c. occasionally. 

d. frequently. 

2. I have used a typewriter. . . 

a. never before. 

b. once ot twice. 

c. occasionally. 

d. frequently. 

3. If you needed assistance, was the test administrator helpful? 

a. yes 

b. no 

c. I did not need assistance 

4. The test questions appeared on the screen. . . 

a. too quickly. 

b. too slowly. 

c. at about the right speed. 
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6. 



7. 



8. 



Overall, the computerized test was. . . 

a. more difficult than a paper-and-pencil test. 

b. easier than a paper-and-pencil test. 

c. about as difficult as a paper-and-pencil test. 

The computerized test was. . • 

a. faster than a paper-and-pencil test. 

b. slower than a paper-and-pencil test. 

c. about the same as a paper-and-pencil test. 

I could read the test questions on the screen. • . 

a. with great difficulty. 

b. with some difficulty. 

c. easily. 

d. very easily. 

The test was confusing. . . 

a. only during the instructions. 

b. only when answering questions. 

c. during both instructions and answering questions. 

d. not at all. 

My eyes felt tired. . . 

a. frequently. 

b. occasionally. 

c. once or twice. 

d. not at all. 
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10. How clear do you feel the computerized instructions were? 

a. Very clear - I had no trouble at all with them 

b. Clear enough, in general - but could be improved 

c. Unclear in places or in part 

d. Very unclear and confusing 

11. Did you have enough time to give your answers? 
a* I didn't feel rushed or pressured at all 

b. I felt a little rushed and could have used more time 

c. I felt rushed and pressured for time 

12. What is your opinion of the difficulty level of the questions? 

a. They were too difficult 

b. They were about right 

c. They were too easy 

13. I understood the test administrator's instructions and introduction to 
the test. 

a. Agree 

b. Disagree 

c. Undecided 

14. I did not need the test administrator's instructions in order to take 
the test. 

a. Agree 

b. Disagree 

c. Undecided 
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15. I did not need the computerized instructions in order to take the test. 

a. Agree 

b. Disagree 

c. Undecided 

lb. I had difficult/ in locating the proper keys on the keyboard. 

a. Agree 

b. Disagree 

c. Undecided 

17. I had difficulty in pressing in right keys. 

a . Agree 

b. Disagree 

c. Undecided 

18. I felt uneasy about taking the test on a computer. 

a. Agree 

b. Disagree 

c. Undecided 

19. The noise from the computer bothered me while I was taking thr, test. 

a. Agree 

b. Disagree 

c. Undecided 

20. Computerized testing is more impersonal than paper and pencil testing. 

a. Agree 

b. Disagree 

c. Undecided 
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21 • I enjoyed taking the test on a computer* 
a* Agree 
b» Disagree 
c. Undecided 

22 • I was bothered by not being able to change my answers at the end of the 
test • 

a* Agree 

b* Disagree 

c. Undecided 

23 . My eyes were tired by the end oi the test, 
a* Agree 

b. Disagree 

c. Undecided 
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